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[57] ABSTRACT 

A system for automatically detecting and recognizing the 
identity of a deformable object such as a human face, within 
an arbitrary image scene. The system comprises an object 
detector implemented as a probabilistic DBNN, for deter- 
mining whether the object is within the arbitrary image 
scene and a feature localizer also implemented as a proba- 
bilistic DBNN, for determining the position of an identify- 
ing feature on the object such as the eyes. A feature extractor 
is coupled to the feature localizer and receives coordinates 
sent from the feature localizer which are indicative of the 
position of the identifying feature and also extracts from the 
coordinates information relating to other features of the 
object such as the eyebrows and nose, which are used to 
create a low resolution image of the object. A probabilistic 
DBNN based object recognizer for determiniog the identity 
of the object receives the low resolution image of the object 
inputted from the feature extractor to identify the object. 

13 Claims, 6 Drawing Sheets 
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NEURAL NETWORK FOR LOCATING AND 
RECOGNIZING A DEFORMABLE OBJECT 

FIELD OF INVENTION 

The present invention relates generally to machine vision 
and more particularly, to a system which implements 
decision-based neural networks that operate to locate and 
recognize deformable objects such as the human face. 

BACKGROUND OF THE INVENTION 

The task of detecting and recognizing a deformable 
pattern or object is an important machine learning and 
computer vision problem. The task involves finding and 
identifying a specific but locally deformable pattern in an 
image, such as a human face. Machine learning and com- 
puter vision has many important commercial applications. 
Such applications include but are not limited to ATM, access 
control, surveillance, and video conferencing. Accordingly, 
machine learning and computer vision has attracted much 
attention in recent years. 

Face recognition systems used in person identification, 
typically employ a face detector which determines the 
location and extent of one or more human faces in a 
non-uniform arbitrary image scene. Such systems find this 
task difficult because human faces are naturally structured 
and made up of deformable components such as the cheeks, 
the mouth, the forehead, etc. In any case, once the face has 
been located, the system then compares the face to other 
faces stored in a database in order to identify the person. 

For systems used in many visual monitoring and surveil- 
lance applications, it is important that the system be capable 
of determining the position of the human eyes from an image 
or an image sequence containing a human face. Once the 
position of the eyes is determined, all of other important 
facial features, such as the positions of the nose and the 
mouth, can be determined. This information can then be 
used for a variety of tasks, such as to recognize a face from 
a given face database 

The key issue and difficulty in face detection is to account 
for the wide range of allowable facial pattern variations in a 
given image scene. In the past, there have been three main 
approaches for dealing with these pattern variations, 
namely: (1) the use of correlation templates, (2) spatial 
image invariants, and (3) view-based ei gen-spaces, etc. 

Correlation templates compute a similarity measurement 
between a fixed target pattern and the candidate image 
location. If the output exceeds a certain threshold, then a 
match is confirmed, i.e., a face detected. There are some face 
detection systems that use a bank of several correlation 
templates to detect major facial subfeatures in an image 
scene. However, the performance of such systems is limited 
in that the class of all potential face patterns is too varied to 
be modeled by a simple bank of correlation templates. 

Spacial image-invariance schemes assume that some 
common and unique spatial image relationships exist in all 
face patterns. Such a set of image invariants must be 
checked for positive occurrences of these invariants at all 
image locations. One particular image-invariance scheme 
for example, is based on the local ordinal structure of 
brightness distribution between different parts of a human 
face. 

Avery closely related approach to correlation templates is 
that of view-based eigen-spaces. This approach assumes that 
the set of all possible face patterns occupies a small and 
easily parameterizable sub-space in the original high dimen- 
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sional input image vector space. Typically, the approach 
approximates the subspace of face patterns using data clus- 
ters and their principal components from one or more 
example sets of face images. An image pattern is classified 

5 as "a face" if its distance to the clusters is below a certain 
threshold, according to an appropriate distance metric. 
However, this approach has only been demonstrated on face 
images in substantially uniform backgrounds. 

There are algorithms and techniques which presently exist 

10 for eye localization are generally based on Hough transform, 
geometry and symmetry check, and deformable models. 
Most of these algorithms and techniques are generally 
inadequate against shape changes, and are time consuming. 
Furthermore, none of these existing methods can locate eyes 

15 when the eyes are closed. 

Neural network models have been found to be very 
amenable to face recognition systems. As is well known in 
the art, a neural network is generally an implementation of 
an algorithm which enables a computer to be adaptive by 

20 learning directly from inputted data which is used to "train" 
the computer to perform a certain task. This enables such a 
computer to process data that only somewhat resembles the 
training data. Moreover, such computers are also capable of 
processing incomplete or imperfect data or providing a 

25 measure of fault tolerance . Additionally, such computers can 
recognize complex interactions among the input variable of 
a system. Since neural networks are parallel, a large network 
can achieve real-time speeds making their application more 
practical in many areas. 

A neural network is generally comprised of intercon- 
nected computational elements or units which operate in 
parallel and are arranged in patterns which broadly mimic 
biological neural networks. Each connection between com- 

35 putational elements is associated with a modifiable weight. 
In operation, a computational element converts a pattern of 
incoming signals into a single outgoing signal that it sends 
to other connected computational elements. It does this by 
multiplying each incoming signal by the weight on the 

40 connection and adds together all the weighted inputs to get 
a quantity called the total input. Then, the computational 
element uses an input-output function that transforms the 
total input into an outgoing signal. In order for the neural 
network to perform a specific task, the computational ele- 

4S ments must be connected to each other in certain network 
arrangement, and the weights must be set appropriately. The 
connections determine how the computational elements will 
influence each other and the weights determine the strength 
of this influence. 

50 It is, therefore, an object of the present invention to 
provide a decision-based neural network and system for 
implementing the network that locates and recognizes 
deformable objects with specific applications directed at 
detecting human faces and locating eyes in the faces. 

55 SUMMARY OF THE INVENTION 

A system for automatically detecting and recognizing the 
identity of a deformable object such as a human face, within 
an arbitrary image scene. The system comprises an object 

60 detector implemented as a probabilistic DBNN, for deter- 
mining whether the object is within the arbitrary image 
scene and a feature localizer also implemented as a proba- 
bilistic DBNN, for detennining the position of an identify- 
ing feature on the object such as the eyes. A feature extractor 

65 is coupled to the feature localizer and receives coordinates 
sent from the feature localizer which are indicative of the 
position of the identifying feature and also extracts from the 
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coordinates information relating to other features of the 
object such as the eyebrows and nose, which are used to 
create a low resolution image of the object. A probabilistic 
DBNN based object recogni2er for determining the identity 
of the object receives the low resolution image of the object 5 
inputted from the feature extractor to identify the object. 

Also provided in the present invention is a method for 
automatically detecting and recognizing the identity of a 
deformable object within an arbitrary image scene. In the 
method, the image scene is preprocessed into subimages. 10 
Each of the subimages are compared with an object detector 
database which stores different versions of the object in 
order to determine whether any of the subimages is the 
object. The coordinates of an identifying feature on the 
object are then located by comparing the coordinates with a 15 
feature localizer database which stores coordinates indica- 
tive of different versions of the identifying feature. Infor- 
mation relating to other features of the object are extracted 
from the coordinates to create a low resolution image of the 
object. Next, the low resolution image of the object image is 20 
inputted into an object recognizer and the identity of the 
object is made. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 

For a detailed understanding of the present invention, 
reference should be made to the following detailed descrip- 
tion taken in conjunction with the accompanying drawings 
wherein: 

FIG. 1 is a diagrammatic view of an exemplary embodi- 30 
ment of the face locating and recognition system of the 
present invention; 

FIG. 2A is a schematic diagram of a DBNN; 

FIG. 2B is a structural depiction of a probabilistic DBNN 
according to the present invention; 35 

FIG. 3 is a schematic diagram of a probabilistic DBNN 
according to the present invention; 

FIG. 4 is a diagrammatic view of second exemplary 
embodiment of the face locating and recognition system of 
the present invention which includes a face verifier; and 

FIG. 5 is schematic diagram of a multi-channel DBNN 
according to the present invention. 

DETAILED DESCRIPTION OF THE 

INVENTION 45 

Although the present invention can be used to locate most 
any deformable pattern or object, the present invention is 
especially suited for use in face detection, eye localization 
and person identification. Accordingly the present invention so 
will be described in this context. 

Face detection, eye localization, and face recognition are 
essentially pattern classification problems. For example, in 
face detection a given pattern is classified into two classes, 
face or non-face. In the present invention, a probabilistic 55 
variant of a decision -based neural network (DBNN) is 
provided to perform this classification task. More 
specifically, both face detection and eye localization are 
implemented by a probabilistic DBNN which will be 
described later on in greater detail. For these applications, 60 
and more generally for any deformable pattern detection, 
there is only one subnet required in the DBNN. In the 
exemplary embodiment of the present invention, the subnet 
represents the face/eye class. Tlius, for an input pattern x, if 
the discriminant function value of the subnet is larger than 65 
the threshold, then x is recognized as a face/eye. Otherwise, 
x is considered as a non -face/eye. 
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Referring now to FIG. 1, an exemplary embodiment of a 
fully automatic face recognition system of the present inven- 
tion is shown and designated by numeral 10. The system 10 
comprises a video camera 12 for inputting an arbitrary 
image scene 11 with 320 by 240 pixels. A DBNN-based face 
detector 14 is coupled to the video camera 12 and includes 
a memory 16 which operates as a database for storing 
images of different human faces. The face detector 14 
determines whether a face is within the arbitrary image 
scene 11. The data stored in the face database 16 is used to 
train the face detector 14. During training, updated network 
weighting parameters and thresholds are stored in the face 
database 16. 

The input images from the video camera 12 are first 
preprocessed before inputting to the DBNN-based face 
detector 14. The inputs to the DBNN-based face detector 14 
are a set of images with predefined coordinates. To detect a 
face in an input image, each of the possible subimages is 
processed to see if it represents a face. A confidence score is 
produced, indicating the system's confidence on this detec- 
tion result. If the score is below some threshold, then no face 
is detected. 

If positive identification of a face is made by the face 
detector 14, a DBNN-based eye localizer 18 which is 
coupled to the face detector 14, is activated to locate both 
eyes in the face image. Knowing the exact position of the 
eyes provides a very effective means for normalizing the 
face size and reorienting the face image. The pattern reso- 
lution used for the eyes is substantially higher than that used 
for the faces. Both the face detector 14 and the eye localizer 
18 are insensitive to small changes in the head size, the face 
orientation (up to approximately 30%), and the presence of 
eye glasses. 

The eye localizer 18 also includes a memory 20 which 
operates as a database for storing information pertaining to 
the coordinates of various different eyes. The eye localizer 
18 determines the coordinates of each eye and then sends 
these coordinates to a facial feature extractor 22 as will be 
explained later below. The data stored in the eye database 20 
is used to train the eye localizer 18. During training, updated 
network weighting parameters and thresholds are stored in 
the eye database 20. 

The facial feature extractor 22 is coupled to the eye 
localizer 18 and uses the eye coordinates sent from the eye 
localizer 18 to extract a low resolution subimage which is 
approximately 140 by 100 pixels and corresponds to the face 
region. The facial region contains the eyebrows, the eyes, 
and the nose (excluding the mouth). Such a facial region 
yields a very high degree of confidence in that it offers 
stability against different facial expressions, hair styles, and 
mouth movement. Improved classification can also be 
gained from secondary facial features such as the hairline 
and the mouth. 

The facial feature extractor 22 normalizes the intensities 
and the edges in the facial region (to a range between 0 and 
1) to compensate for changing illumination. Edge filtering 
and histogram modification techniques can be applied to 
recondition the facial images. The normalized and recondi- 
tioned images of 140 by 100 pixels are then reduced to 
coarser feature vectors of approximately 13 by 9 pixels. 
Adopting lower resolution fecial features provides substan- 
tial reductions in computational cost and storage space and 
increases the tolerance on face/eye location errors. 

In order to assure sufficient diversity of real facial images 
in the training set, the algorithm takes the acquired sensor 
image and transforms it to create additional training exem- 
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plars otherwise known as virtual training patterns. Two 

kinds of training patterns are used. The first training pattern (m 

consists of positive patterns (face/eye patterns) which are ^ * w t m ^^ m > w t W (J) 

used for reinforced learning. The second training pattern Accordingly, the winning class for the pattern is the jth 

consists of negative patterns (non-face/eye patterns) which 5 class ( SUD net). When and only when j*i , (i.e., when x (m) is 

arc used for antireinforced learning. The network weighting misclassified), the following update will be performed: 
parameters and the thresholds are updated by this reinforced/ 

antireinforced learning. A more detailed discussion of virtual Reinforced Learning: w^W/^+n V>fc h£ 

pattern generation will be explained later below. Antireinforced Learning: */^*/">^V<K* „,) (2) 

The feature vector generated by the facial feature extrac- *0 

tor is then fed into a DBNN-based face recognizer 24 for Typically, one output node is designated to represent one 

recognition. The face recognizer 24 includes a memory 26 class. The All-Class-in-One-Network (ACON) structure is 

which operates as a database for storing person identification adopted by in a conventional MLP, where all the classes are 

information. lumped into one super-network. The supemet has the burden 

The trained system can be easily adapted to a face 15 ° f »»ving ,'° ^simultaneously satisfy all the teachers, so the 

verification application. Due to the distributed structure of a number of hidd^enumtsK tends to be large topmcal results 

DBNN, any individual person's database may be individu- have confirmed that the convergence rate of ACON degrades 

ally retrieved for verification of his or her identity as drastically with respect to the network size because the 

proclaimed training of hidden units is influenced by potentially conflict- 

The DBNN techniques implemented in the face detector 20 "Jg signals &om different teachers. . 

^. iL i i- i© j *u r -ia Referring to FIG. 2 A, a schematic diagram showing the 

14, the eye localizer 18 land the face recognizer 24 as fjbnN Ja the pre sent invention designated by the 

described in the system 10 above can be applied in other ^ ^ ^ ^ DBNN M ^ ^ 0ne ^ lass . 

sunilar systems for detecting virtually any type of deform- . _ . . . m ' vr . t , , , . 

t_i i_- * j- • i_- i_ c ii • » . . * m-One-Network (OCON) structure, where one subnet is 

able obiect. The discussion which follows immediately ,. . 4 v t ' - r « u * *>i j <%c 

. , j « . ... . . t^v^t . , t , . 25 designated to one class only. Each subnet 32, 34 and 36 

below details the probabilistic DBNN implemented in the * . ,. . . , i t c / u 4 « 

above described s stem specializes in distinguishing its own class from the others, so 

' j mc number of hidden units is usually small. Experimental 

The DBNN used in the present invention uses a distnb- results based on a bfoad range of appucat ions (OCR, speech, 

uted and localized updating rule based on reinforced and and face recogoition) suggest mat 3^5 mddeQ ^ 

anti-reinforced learning strategy. The gradient of the dis- 3Q subnet m pre f erred . The OCON structure of a DBNN 

criminant function with respect to the weight parameters is makes - t mQSt suilable for incremental u&iningf Le ., network 

used as an updating direction. The main ment of this is that upgrading upon add ing/removing memberships, 

it enables the border between any two classes to be settled The ttaini of the DBNN 30 fc based on hxaSLy 

mutually, with minimum side-effects on other borders. In the UnsU p e rvised Globally Supervised (LUGS) learning. There 

DBNN, the teacher only tells the correctness of the classi- 3S afC tWQ phases in this scheme: during the i OC ally- 

fication for each Gaining pattern. The teacher is a set of uasuperviscd (LU) phase , each subnet is trained 

symbols, x={t,-}, which label the correct class for each input ^vi^y, and n0 mutua l information across the classes 

pattern. Unlike an approximation formulation, exact values may ^ utilizcd MiCT the Lu phase fa the 

of the teachers are not required. Accordingly the objective eQters ^ GJobally _ Supervised (GS ) phase. In GS 

of the training is to find a set of weights which yields a 4Q phase teacher information ^ introduced to reinforce or 

correct classification. anti-reinforce the decision boundaries obtained during LU 

For complex pattern distribution, the discriminant func- phase. The discriminant functions in all clusters will be 

tion is usually a priori unknown. This leads to a credit trained in a two-phase learning technique comprising a 

assignment rule on when, what, and how to perform network global level and a local level. In the global level phase of 

updating. Its main purpose is to alleviate the problem of 4S learning, a supervised mutual (decision-based) learning rule 

overtraining the networks. Tbere are three main aspects of ^ adopt ed. In the local level phase of learning, initialization 

the training rule: when to update; what to update; and how ^ ai ways Dy an unsupervised clustering method, such as a 

to update the weights. k-mean. If too many clusters are adopted, overfitting can 

Under the training rule, knowing when to update is result, which in turn will hamper the generalization capa- 

determined by a selective training scheme for example, 50 bility. A proper number of clusters can be determined by an 

which updates the weight only when there is misclassifica- unsupervised clustering technique. 

tion. Since the rule is distributive and localized, knowing The learning rule of the DBNN 30 is very much decision- 

what to update is accomplished by applying reinforced boundary driven. When pattern classes are clearly separated, 

learning to the subnet corresponding to the correct class and such learning usually provides very fast and yet satisfactory 

anti-reinforced learning to the subnet corresponding to non- 55 learning performance. Application examples including OCR 

correct class. Updating under the rule is accomplished by and (finite) face/object recognition. Different tactics are 

adjusting the boundary by updating the weight vector w needed when dealing with overlapping distribution and/or 

either in the direction of the gradient of the discriminant issues on false acceptance/rejection, which arises in appli- 

function (i.e., reinforced learning) or opposite to that direc- cations such as face recognition and verification. In such 

tion (i.e., antireinforced learning). 60 applications, the present invention provides a probabilistic 

The following describes the Decision-Based Learning variant of the DBNN as described earlier in connection with 

Rule discussed immediately above: Suppose that S={x (1 * . the face detector 14, eye localizer 18 and face recognizer 24 

. . , x w } is a set of given training patterns, each correspond- of the automatic face recognition system 10 of FIG. 1. 

ing to one of the M classes {co l7 i=l, . . . , M}. Each class is Referring to FIG. 2B, an exemplary embodiment of a 

modeled by a subnet with discriminant functions, for 65 probabilistic DBNN denoted by the numeral 40 is schemati- 

example, <|>(x,w J i=l, ... ,M. Suppose that the m-th training cally shown. The subnets 42 and 44 of the probabilistic 

pattern x (,n) is known to belong to class co, and; DBNN 40 are designed to model log-likelihood functions. In 
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the probabilistic DBNN 40, reinforced/antireinforced learn- bution with full-rank covariance matrix. A hyper-basis func- 

ing is applied to all the clusters of the global winner and the tion (HyperBF) is meant for this. However, for those appli- 

supposed (i.e. the correct) winner, with a weighting distri- cations which deal with high dimensional data but finite 

bution proportional to the degree of possible involvement number of training patterns, the training performance and 

(measured by the likelihood) by each cluster. 5 storage space discourage such matrix modelling. A natural 

The probabilistic DBNN 40 is designed to approximate simplifying assumption is to assume uncorrelated features cf 

the Bayesian posterior probabilities and likelihood func- unequal importance. That is, suppose that p(I|co ( ., GJis a 

tions. It is well known that the optimal data classifier is the D-dimensional Gaussian distribution with uncorrelated 

Bayes classifier. Suppose there are M classes {to,, . . . , (a M } features, that is 

in the feature space, the Bayes decision network classifies 1Q * 

input patterns based on their posterior probabilities: Input x / \ (4) 

is classified to class w f if P(o>Jx)>F(a>jlx), for all j*i. It can 1 / _ j_ f - M 2 \ _ 

be shown that Bayes classifier has minimum error rate. " D \ 2 d" 1 ) 

Suppose the likelihood density of input x given class (o t (2ji) W2 11% \ ' 

is a D-dimensional Gaussian distribution, then posterior d 

probability P(coJx) can be obtained by Bayes rule: 15 whcrc o>^, . . . , co^] r is the mean vector, and 

diagonal matrix 

where P(co i ) is the prior probability of class co ; (by definition 20 is tte covariance matrix. 

To approximate the density function in Eq. 4, we apply the 

M M elliptic basis functions (EBF) to serve as the basis function 

2 p(o)i)-i),andpC)- 2 i>(oO/<^jO for each cluster: 

t=i Jfc=i 

25 1 D 1 (*) 

The class likelihood function p(x|co t ) can be extended to °*> e ') 2 ^ ~7~ ^ ~ Wrd ^ + Qr 

mixture of Gaussian distributions. Define p(x|co / , 0 r ) to be "* 
one of the Gaussian distributions which comprise p(x|co t ), whcre 



p{^) - S P(e^(*K 0 r ) 30 8,--— - In2» - 2 lnow. 

r-1 L «=1 



where B T ={^, denotes the parameter set for the cluster 
t, P(©Jco i ) denotes the prior probability of cluster x when 
input patterns are from class (n i7 and p^u)^ ©^N^, XJ. 
By definition 

r-1 

In many applications it is appropriate to assume that F^go,) 
-P(a)j). Therefore the likelihood probability p(x|aj J ) can 
replace posterior probability F(co,|x) to serve as discriminant 
function. 

The discriminant function of each subnet in the probabi- 
listic DBNN models the log-likelihood function: 



<K* m) - logp(*1<»;) - log I 2 Pipfadpixfcr, a),) j 

The overall diagram of such a discriminant function is 
depicted in FIG. 2B which shows the structure of the 
probabilistic DBNN. The function node f( ) is a nonlinearity 
operator. If we make the assumption of P(w i )=P(co y ), f() is 
a log operator (likelihood type). If P(o3 1 -)^P(co > , /( ) is a 
normalization operator. Its purpose is to make the discrimi- 
nant function approximate class posterior probability 
(posterior type). The DBNN shown in the exemplary 
embodiment is of the likelihood type. Note that an explicit 
teacher value would not be required, although it is a super- 
vised training because the teacher's knowledge on correct 
classification is crucial in the training. In FIG. 3, a schematic 
diagram of the probabilistic DBNN 40 for deformable object 
detection is shown. The action described in the parentheses 
is for the learning phase only. 

In the most general formulation, the basis function of a 
cluster should be able to approximate the Gaussian distri- 



Afler an exponential transform, exp {to(x, to t -, ©,)} can be 
viewed the same Gaussian distribution as described in Eq. 4, 
35 except a minor notational change: l/o^-a,/. In other 
words, 

1 o (6) 

V{X, 0),, 6,) 2 O^Xd-wJf + dr 

2 d=l 

40 

The learning rules for the probabilistic DBNN 40 is as 
follows. In the training scheme for the DBNN the LUGS 
principle is followed. The Locally Unsupervised (LU) phase 
for the probabilistic DBNN can adopt several unsupervised 
45 learning schemes such as, LVQ, k-mean, EM as so forth. As 
for the Globally Supervised (GS) learning, the decision- 
based learning rule is adopted. Suppose that the m-th train- 
ing pattern x* m) is known to belong to to,-; and 

so ¥^ m \ w/" 0 *^" 0 , wf m >), vi*j CO 

That is, the winning class for the pattern is the j rt class 
(subnet). When and only when j*i, (i.e., when x* m) is 
misclassified), the following update will be performed: 

^ 5 Reinforced Learning: wf^^-wW-vfiVtfx, wj) 

Antireinforced Learning: n/^W/^-tiV+fc *>j) (8) 

If the training pattern belongs to the so-called negative 
60 training (i.e. "unknown") set, then only the anti-reinforced 
learning rule will be executed — since there is no "correct" 
class to be reinforced. 

The gradient vectors in Eq. 8 are computed as follows: 

— M*) ■ - *w) 
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-continued tem ^ be included in the negative training set. When 

training the probabilistic DBNN, positive patterns are used 

j_ / _j w . 2 \ for reinforced learning, and negative patterns are used for 

*M X ) 2ya^" / anti-reinforced learning. 

5 The third aspect is the generation of run-time negative 
pattern. During the training phase, the probabilistic DBNN, 

P(9Av>dp(4at, ©r) Wnile stiU unclcr tramm g, 0411 ^° uscd t0 cxam i ne tne whole 

A ^ " s f(8>kl)p(^tl Qt) image database every k epochs. If the network falsely 

*' detects a face (eye) somewhere in an image, then that 

particular subimage will be included into the negative train- 

F^ejco^) (and P((D f ), if P(co i )rfP(co / )) can be learned by the 10 mg ^ 

EM algorithm: At epoch j, Since the probabilistic DBNN is amenable to multi sensor 

classification, additional sensor information can be easily 

&) W&M'ty&Mi* Q r) (10) incorporated to improve the recognition performance. Two 

r ~ lj^B^<a^p(x^ci h ©t) approaches to multi-sensor classification are possible in the 

15 present invention. The first approach involves a hierarchical 

pfejtt-W+i) - fi/AO I k®(m) classification, where sensor informations are cascaded in 

I H^J -K m~i sequential processing stages. Possible candidates for hier- 
archical sensor are the hairline and the mouth. The second 

Pfo) - nm 2 Pfto.W'-))© approach involves a multi-sensor fusion, where sensor infor- 

m-a 20 mations are laterally combined to yield improved classifi- 
cation. 

Since the probabilistic DBNN 40 provides probabilistic Referring to FIG. 4, « t hierarchfcal^rmation processing 

. , . 4 . „ r „ r . system based on probabilistic DBNN is shown denoted by 

outputs a similar procedure to the Neyman-Pearson hypoth- ^ ^ * 50 fc similaf tQ ^ of R J 

esis is followed for threshold updating. Accordingly, testing ± ^ include& a preprocessing module 5 2 which 

is performed by setting a threshold to the network outputs provides hairline or mouth features. In FIG. 4, hairline 

and computing the probability of false acceptance and false features are provided by the preprocessing module 52. The 

rejection. In order to find out the most possible regions for hairline images are inputted to face verifier 54 along with 

patterns from class co,, it is preferred to choose a threshold decisional information provided by the face recognizer 24. 

T £ so that an input x is classified to class w • if log p(x|a) £ )>T t -. Generally, this system operates by cascading two processing 

For an input x, if xeco t - but log p(x|co £ )<T i , then T, should 30 stages. More specifically, a face verifier stage is cascaded 

reduce its value. On the other hand, if xgoo, but log after the (original) face recognizer stage. The face verifier 52 

p(xjco I -)>T l , then T ( should increase. An adaptive learning is itself is another DBNN classifier. Its function is to 

rule to train the threshold. T, is proposed in the present verify/reject the decision of the primary recognizer. The 

invention as follows: Define daT-log p(xjco). Also define a verifier can be trained by the decision-based learning rule, 

cost function 1(d). 1(d) can be either a step function, a linear 35 The input vector is a 12x8 feature vector obtained by 

function, or a fuzzy-decision sigmoidal function. Once the down-sampling the forehead/hairline region of the face 

network finishes the training, the threshold values can be image. Three verification schemes are possible in this sys- 

trained as follows: Given positive learning parameter tj, at tem - , , „ t t 

st • In the first scheme, recall that each subnet of the primary 

40 DBNN recognizer generates a confidence score for an input 

T^~T^-y\i\d) if xGtoj (reinforced learning) pattern. Suppose that the highest scorer is the i-th subnet. If 

the confidence score of subnet i is below the threshold, the 

Tp+o-TP-nn*) if (antircinforced learning) (11) top choice of ^ face ver ifi er is checked. If the best match 

The foUowing discussion details the technique used for of foreheaaVhairline region is also class i, then class i is 

generating training patterns for the probabilistic DBNN. 45 recognized land verified. Otherwise the test pattern is deemed 

Generally! there are three main aspects to the training pattern £ & ' a ^ f ,afidenCC 80016 ° f * ^ 

4 . J , * * ,i 4 . . . ° r the threshold, and if class l is among the top k (e.g. k=6) 

generate scheme used m the present mvention matches Qf ^ forehead/hairline ^ ^ y KC ^ iion £ 

The first aspect is the generation of virtual training otherwise the person is rejected. 

patterns. In the beginning of the training phase, a certain Tq tfac scheme, similarity fists are introduced. 

number of facial images are selected to produce exemplar 50 £very clafiS hflS {t& own similarity ^ nc list's lengths also 

face/eye patterns for training the earlier described DBNN- vary from person t0 person. Initially, the similarity list of 

based face detector and eye localizer. Typically, these exem- c i ass j contains only class j itself. Assuming that the DBNN 

plar face/eye patterns can be extracted manually from those f ace verifier has now completed the training process by the 

images. For each exemplar pattern, virtual patterns can be decision-based learning rule. If a training pattern (originally 

generated by applying various affine transformations such 55 from class j) is classified into another class, say, k, then the 

as, rotation, scaling, shifting and mirroring process to the class k will be added to the similarity list of class j. This 

original pattern. By this method, each of the exemplar process will be repeated until all the training patterns of the 

patterns is used to regenerate a number of up to 200 virtual known persons are tested. 

training patterns. In regard to the verification rule, if the confidence score 

The second aspect is the generation of positive/negative 60 of subnet i is below the threshold, the top choice of face 

training patterns. Not all virtual training patterns are con- verifier is checked. If the best match of the forehead/hairline 

sidered good face or eye patterns. If a virtual pattern is region is also class i, then class i is recognized and verified, 

slightly perturbed from the original exemplar pattern, it will Otherwise the test pattern is deemed as an intruder's. If the 

be included in the positive training set. This generally confidence score of subnet i is above the threshold, and if the 

enhances the robustness of the neural network. On the other 65 top one class of the face verifier is on the similarity list of 

hand, if the perturbation exceeds a certain threshold class i, the recognition is confirmed. Otherwise the person is 

(empirically established by trial-and-error), the virtual pat- rejected. 
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The third scheme is the same as the second scheme, At step j, 
except the confirmation is now made more strict. More 

specifically, if the confidence score of subnet i is below the ^ (m) m ojVfrWfoCO ^ _ ^ £ ^ (12) 

threshold, its recognition may still be confirmed if, the top x, o^p^^ ci) 

choice of the face verifier is class i and its confidence score 5 ^ ^ m is trained, then the fusion weights will 

exceeds the threshold of the verifier. Otherwise the person remam C0QStan( during me retrieving phase . 

will be rejected. If the confidence score of subnet i is above A more ge neral version of multi-channel fusion, referred 

the threshold, the recognition can be confirmed, only if the t0 ^ data-dependent channel fusion is also presented. 

top choice of the face verifier is on the similarity list of class io Instead of using the likelihood of observing x given a class 

i and its confidence score exceeds the threshold of the (p(xja)„ C^)) to model the discriminant function of each 

verifier. Otherwise the person will be rejected. cluster, we shall use the posterior probabilities of electing a 

Al1 4 , c , , ( „ • „ class given x (p(cu / |x,C K )). For this version of the multi- 

All three of these schemes substantially improve the , & \r\ n > 

r - r ■ - * i w u channel network, a new confidence P(CJx) is assigned, 

performance of face recognizer. Experimental results have ^ M ^ fof ^ oonfidcnoc on ^ hen mc input 

shown that the third scheme produces about the same (false pat|em fc x Accordingly) ^ probability model is also 

acceptance+false rejection) rate as the second scheme does. modified to become 

The difference between these two schemes is that the false 

rejection rate of the third scheme is higher while the false K (12A) 

acceptance rate is lower. 20 *=i 

Referring to FIG. 5, an exemplary embodiment of a p(a) j x> c ^ can ^ obtained by P(to j x> c^cojC^xK 

multi-channel DBNN for multi-sensor biometric recognition C^/p^Ot), and the confidence P(Cjx) can be obtained by 

is shown and denoted by the numeral 60. This approach me following equations: 
consists of several classifier channels C1,1-C1,6 and C2,l- 25 

C2,6 each of which receives input vectors either from rjcjfc) - p ^ c ^ p ^ c ^ ^ 12B ^ 

different sensors or from a portion of a higher dimensional %tP(Ci)p{x\ci) 

feature vector. Here, the channels C1,1-C1,6 and C24-C2,6 where p (^Q h ) can be computed straightforwardly by equa- 

are not differentiated into primary or secondary categories. ^ on ^^C^ 

Therefore, lateral fusion of information is more appropriate. 30 

The outputs of the channels are combined by some proper ^(a^jc^*!^ 

weightings W11-W26. The weighting factor is assigned , _ . , , , / 1 ^ \ *u 

i_ j .u «j *u a- u u - t and P(C t ) can be learned by 12 (but replace pfxko,, C *) with 

based on the confidence the c^^^^^i^ ^V** tcrrn P(Q ) / aa * ^ interpreted as /"the general 

recognition result Since DBNN generates probabihsUc confidence" we have on channel k. 

outputs, it is natural to design the channel weightings to have 35 ^ ^ appfoach> ^ weights 

probability properties. C fc is the output of the i-th subnet in Qeed {Q ^ computed for each ^Xwg paltern the 

channel k, which is equal to p^a^, C*). Further note that the retrieving phase, 
confidence measure is W Jb =P(Cja) I ) and that the combined 

output for subnet i is Oi, which is p(x|a)J. TEST RESULTS 

In class-dependent channel fusion, the weighting factors Experimental test results are briefly summarized below, 
correspond to the confidence P^oo,) for each channel. j^e probabilistic DBNN has consistendy and reliably deter- 
Here PO^Ju)^ represents the indicator on the confidence on mined actual face positions, based on experiments per- 
channel k when the test pattern is originated from the w, formed on more than one thousand testing patterns. The 
class. (By definition, 45 probabilistic DBNN also yields very satisfactory eye local- 
ization performance. It is insensitive to small changes of the 
* head size, the face orientation (up to approximately 30%), 
^ P(C^G>i) = 1 and the p resence 0 f eve glasses. The present invention is 

very robust against large variation of face features and eye 

50 shapes. The probabilistic DBNN takes only 200 ms on a 

so it has the property of a probability function.) Suppose that S(JN Sparc2 o workstation to find human faces in an image 

there are K channels in the subnet w„ and within each 320x240 pixels. For a facial image with 320x240 

channel there are R clusters. The probability model of the pixels, the probabilistic DBNN takes 500 ms to locate two 

DBNN-based channel fusion network can be described as eyes on a SUN Sparc20 workstation. Furthermore, because 

follows: 5 5 of the inherent parallel and distributed processing nature of 

DBNN, the technique can be easily implemented via spe- 

/ . ^ £ nr. 1 \ 1 j ^ x ^ 11A ) cialized hardware for real time performance. 

A-i The following is an example of the application 

performance, which was based on the experimental perfor- 

where p(x|co^ C*>is the Discriminant function of subnet i in 6Q m&ncGS on public (FE RE T) and in-house (SCR) databases, 

channel k, and p(x|co £ ) is the combmed disenmmant function First> ^ expe riment was conducted on 200-person (each 

for class o t -. Note that x^x/, . . . , x K T f, and since p(xK> with two front- views) of the ARPA/ARL FERET database. 

Q) is conditional on only x* is involved in the above One image per person was used for training and the other for 

formula. After all the parameters within channels complete testing. Adecision-boundary driven DBNN reached 100% in 

their training, channel confidence P^Jo^) can be learned by 55 training accuracy and 96% in testing accuracy. An improved 

the following: Define a^P^co,). At beginning, assign probabilistic variant of the DBNN achieved 99% recognition 

a^l/K, Vk=l , K. rate. The SCR 80x20 database consists of 80 people of 
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different races, ages, and genders. The database contains 20 
images for each person. If a person wears glasses, 10 of the 
images are with glasses and 10 are without glasses. The 
training set comprised 4 images per person. The testing set 
included 16 images per person, 1280 images in total. For all 
of the images, the DBNN-based face detector always cor- 
rectly detected the center of the face thereby providing a 
100% success rate. Eye localization is a more difficult task 
than face detection, in particular when eye glasses are 
present. Among the 1280 images, the DBNN-based eye 
localizer mis-detected the eyes in 5 images by errors of more 
than 5 pixels. For the remaining 1275 images, the DBNN- 
based face recognizer achieved 100% recognition rate. An 
SCR-IM 40x150 database offered an opportunity to experi- 
ment with much larger orientation and other variations. The 
database contained 150 images for each of 40 persons. The 
images were acquired continuously while the person slowly 
moved and rotated his head. Head rotations were not only by 
very wide angle (up to 45 degrees) but also along various 
axes (i.e., left-right, up-down, and tilted rotations). The 
DBNN-based face detector and the DBNN-based eye local- 
izer worked correctly for 75% of the 6000 images in this 
database, which formed the so-called valid data set. A prior 
art face detector and eye localizer were trained only on 
frontal views. They could handle images up to only 30 
degree rotations. The DBNN-based face recognizer 
achieved a very high 98% recognition rate. 

The hierarchical DBNN-based face recognition system 
was tested with a 38-person face database. The hierarchical 
classification significantly reduces the false acceptance from 
9.35% to 0% and the false rejection from 7.29% to 2.25%, 
as compared to non-hierarchical face recognition. 

It should be understood that the embodiments described 
herein are merely exemplary and that a person skilled in the 
art may make many variations and modifications to these 
embodiments utilizing functionally equivalent elements to 
those described herein. Any and all such variations or 
modifications as well as others which may become apparent 
to those skilled in the art, are intended to be included within 
the scope of the invention as defined by the appended 
claims. 

We claim: 

1. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for deterrmning whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a decision-based neural network; 



wherein said decision-based neural network comprises a 
one-class-in-one network structure having a plurality of 
subnets and a plurality of classes, wherein each one of 
said subnets is designated to one of said classes in order 
to distinguish it from said other classes; and 
wherein said decision-based neural network includes a 
training scheme having a first phase and a second 
phase, wherein said first phase includes individually 
training each of said subnets without mutually 
exchanging information between said classes and said 
second phase includes reinforcing learning and anti- 
re inforcing learning obtained during said first phase. 
2. The system according to claim 1, wherein said 
decision-based neural network comprises a probabilistic 
15 decision-based neural network, said reinforcing and anti- 
reinforcing learning being provided by a training pattern 
X< m > belonging to a class and: 
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wherein reinforced learning is performed according to: 
and antireinforced learning is performed according to: 

3. The system according to claim 1, wherein said 
decision-based neural network comprises a probabilistic 
decision-based neural network that includes a plurality of 
probabilistic outputs, each of which have a threshold value 
which is trained according to an adaptive learning rule: 

TF+V-T&-r\l'(d) i£xea>i (reinforced learning) 
TP MV >=7P^t\V{d) if x*o>{ (antireinforced learning). 

4. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of subnets, each of said subnets 
having a plurality of cluster basis functions which 
include cluster prior probabilities according to: 
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5. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of subnets, each of said subnets 
includes a plurality of elliptic basis functions according 
to: 
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6. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of subnets, each of said subnets 



including a discriminant function which comprises a 

nonlinearity operator. 
7. The system according to claim 6, wherein said dis- 
criminant function comprises a log operator which approxi- 
mates a log-likelihood function: 
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8. The system according to claim 6, wherein said dis- 
criminant function comprises a normalization operator 
which approximates a class posterior probability. 

9. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision -based neural 
network; and 

further comprising object verifier means implemented as 
a probabilistic decision-based neural network for veri- 
fying the decision of said object recognizer means, said 
object verifier receiving additional information about 
said object which is cascaded in sequential processing 
stages in a hierarchical manner by said object verifier. 

10. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 
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wherein said probabilistic decision-based neural network 
comprises a plurality of classifier channels each having 
an output, said outputs being laterally fused by weight- 
ing said channels. 

11. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of classifier channels each having 
an output, said outputs being laterally fused by class- 
dependent channel fusion according to: 
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receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of classifier channels each having 
an output, said outputs being laterally fused by data- 
dependent channel fusion according to: 
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12. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 40 
scene, said system comprising: 

object detector means for deteimining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 45 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 50 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 55 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 



13. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature localizer 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object, 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a decision-based one -class-in-one neu- 
ral network structure having a plurality of subnets and 
a plurality of classes, wherein each one of said subnets 
is designated to one of said classes in order to distin- 
guish it from said other classes, wherein said neural 
network includes a training scheme having a first phase 
and a second phase, wherein said first phase includes 
individually training each of said subnets without 
mutually exchanging information between said classes 
and said second phase includes reinforcing learning and 
anti-reinforcing learning obtained during said first 
phase. 
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